WO 2004/051451 



02JUN2005 

PCT/IB2003/005499 

TO/537127 



GRAPHIC USER INTERFACE HAVING TOUCH DETECT ABILITY 



This application is a continuation-in-part of copending U.S. patent application Serial No. 
10/174,284, filed on June 18, 2002. 
5 The subject invention relates to control panels having a flat panel graphic user interface 
(GUI). The invention further relates to a remote control having such a control panel. 
Consumer electronics devices are often provided with remote control units for controlling 
the device from the comfort of one's easy chair. These remote control units typically are 
rectangular in shape and carry a plurality of buttons for operating the various control 

1 0 functions of the electronic devices. 

However, as the user acquires more and more devices, these remote control units start to 
accumulate leaving an array of remote control units on the user's coffee table with the 
problem of selecting the appropriate remote control unit for controlling the desired device. 
This problem has been addressed with universal remote control units which either include 

15 or may be programmed with the codes for controlling a plurality of different devices in a 
plurality of different device categories. As such, a user of such a universal remote control 
unit may control his/her stereo system, television receiver, video cassette recorder, DVD 
player, CD player, cable box, satellite receiver, etc., using the one universal remote control 
unit 

2 0 Quite naturally, it has now become a feat to design such a universal remote control unit 
which can be intuitively used to control all of these devices. 

U.S. Patent 5,956,025 discloses a remote control unit with 3D organized graphic user 
interface (GUI) for a home entertainment system which includes a GUI in the form of, for 
example, a liquid crystal display (LCD) with touch sensitivity in which various icons are 

2 5 displayed on the GUI and represent various control functions which the user selects in 

order to operate the various control functions of the various devices. 
Figs. 1 and 2 show an example of a remote control unit having such a GUI in the form of a 
touch sensitive LCD, which is marketed by Philips Electronics. As shown in Fig. 1 , this 
remote control unit 10 includes an infrared transmitter for transmitting infrared control 

3 0 signals to the devices to be controlled. The remote control unit 10 includes a plurality of 

hard switches including a "MUTE" button 14, channel "UP" (16) and "DOWN" (18) 
buttons, volume "UP" (20) and volume "DOWN" (22) buttons, "RIGHT" (24) and "LEFT" 
(26) buttons for moving a cursor, and a "BACKLIGHT" button (28). The remote control 
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unit 10 further includes a GUI in the form of a liquid crystal display 30 which displays 
various control icons for the device to be controlled. As shown in Fig. 1, the controlled 
device is a television receiver and the control icons include, e.g., "ON" (32) and the 
number"!" (34). 

5 Fig. 2 shows the remote control unit 10 of Fig. 1, now set to control a DVD player, and has 
various control icons, e.g., "ON" (32), "REWIND" (36), and "PLAY" (38). 
However, one problem with these types of remote control units is that it is necessary for 
the user to look at the remote control unit in order to operate it. While this may not be a 
problem when using the remote control unit to operate an audio device, such as, a CD 

1 0 player, when the user is watching television, in many cases, the illumination in the room is 
dimmed to enhance the picture. As such, it is then difficult to discern the markings on the 
display of the remote control unit. While, for example, the remote control unit 10 of Figs. 1 
and 2 includes the button 28 for activating a back light for the display, the user still must 
take his/her eyes off of the television in order to reliably operate the remote control unit. 

1 5 There exists a need for an improved remote control device to overcome these and other 
shortcomings. 

It is an object of the invention to provide a control panel for a device which addresses the 
aforementioned and other shortcomings in the related art. 
A first embodiment of the invention is a control panel for a device, comprising: 
20 a graphical user interface (GUI) displaying a plurality of control icons, wherein the 
plurality of control icons represent a plurality of corresponding control functions for 
controlling the device; and 

means for providing tactile detectability to said GUI to allow a user to detect at least one of 
the plurality of control icons by touch. 
25 A second embodiment of the invention is a method comprising: 

providing a control device including a display for displaying a graphical user interface 
(GUI); 

displaying on the GUI a plurality of control icons representing various control functions 
wherein the control functions enable a user to control a system through the GUI; and 
3 0 adapting at least one of the plurality of control icons so as to be detectable by a user via 
means selected from the group of vibrotactile means, electroctactile means, and 
combinations thereof. 
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With the above and additional objects and advantages in mind as will hereinafter appear, 
the subject invention will be described with reference to we accompanying drawings, in 
which: 

Fig. 1 shows a plan view of a remote control unit having a graphic user interface (GUI) 
5 showing control icons for a television receiver in the related art; 

Fig. 2 shows a plan view of the remote control unit of Fig. 1 in which me GUI shows the 
control icons for a DVD player; 

Fig. 3 shows a plan view of a GUI portion of an embodiment, in accordance wilh the 
present invention; 

10 Fig. 4 shows an edge view of the GUI of the remote control unit of Fig. 3 incorporating a 
first embodiment of the subject invention; 

Fig. 5 shows an edge view of the GUI of the remote control unit of Fig. 3 incorporating a 
second embodiment of the subject invention; 

Fig. 6 shows an array of actuators and a block diagram for controlling the actuators. 
15 Fig. 7 shows a side sectional view of a control panel with a vibrotactile display, in 
accordance with the present invention; 

Fig. 8 A shows a side sectional view of a portion of a control panel with an electrotactile 
display, in accordance with the present invention; 

Fig. 8B shows a side sectional view of a control panel with an electrotactile display, in 
2 0 accordance with the present invention; 

Fig. 9A shows a plan view of a portion of a centralized-type vibrotactile display, in 
accordance with the present invention; 

Fig. 9B shows a plan view of a portion of a distributed-type vibrotactile display, in 
accordance with the present invention; 

2 5 Fig. 9C shows a plan view of another embodiment of a portion of a distributed-type 

vibrotactile display, in accordance with the present invention; 

Fig. 10A shows a plan view of a centralized-type vibrotactile display depicted a TV 

display interface, in accordance with the present invention; 

Fig. 10B shows a plan view of a centralized-type vibrotactile display depicted a VCR 

3 o display interface, in accordance with the present invention; 

Fig. 10C shows a plan view of a distributed-type vibrotactile display depicted a VCR 
display interface, in accordance with the present invention; 
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Fig. 1 1 A shows a plan view of an embodiment of a control panel depicting a TV display 
interface, in accordance with the present invention; 

Fig. 1 IB shows a plan view of an embodiment of a control panel depicting a TV display 
interface starting a transition to a VCR display interface, in accordance with the present 
5 invention; 

Fig. 1 1C shows a plan view of an embodiment of a control panel depicting a TV display 
interface further transitioning to a VCR display interface, in accordance with the present 
invention; and 

Fig. 1 ID shows a plan view of an embodiment of a control panel depicting a VCR display 

1 0 interface, in accordance with the present invention. 

Fig. 3 shows the GUI 50 of a remote control unit incorporating the subject invention. The 
GUI includes three regions, an information section 52 which includes information for the 
selected device, a device selection section 54 for displaying control icons for selecting the 
desired device, e.g., TV 58 and LD (laser disc) 60, and a control function section 56 for 

1 5 displaying control icons for controlling the desired device, in this case, numbers 1-0 icons 
61-70. 

Fig. 4 shows an edge view of the GUI 50 where it will be apparent that surface of the GUI 
in the vicinity of the icons 62, 64, 66, 68 and 70 is raised. It should be understood that 
while Fig. 4 only shows some of the icons being raised, the surface of the GUI in the 
2 0 vicinity of other icons, including 58, 60, 61, 63, 65, 67 and 69, may also be raised. 

Once a user of the remote control unit 50 is familiar with the layout of the control icons, 
the user is then able to select the appropriate icon by merely sliding his/her finger across 
the surface of the GUI thereby detecting the raised areas and then selecting the desired 
control icon represented by the appropriate raised area. 

2 5 While this embodiment of the invention allows for a user to discern the various control 

function by touch, when the raised areas are formed in the surface of the GUI, the layout of 
the control icons on the display may not be changed. 

Fig. 5 shows a second embodiment of the invention in which the touch sensitive display is 
a flexible display. U.S. Patent 6,368,730 discloses an electroluminescent device which is 

3 0 flexible and may be used for the display in the GUI 50\ The raised portions 62\ 64*, 66', 

68' and 70 axe formed by actuators 72.0-72.9 arranged beneath the flexible display 50'. 
Each of the actuators 72.0-72.9 includes a pusher rod 74.1-74.9 which, upon activation of 
the respective actuator, presses on the under-surface of the flexible display 50\ Hence, in 
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order to accommodate the icons forming the numbers "2", "4", "6", "8" and "0", as shown 
in the GUI 50 of Fig. 3, the actuators 72.1, 72.3, 72.5, 72.7 and 72.9 are activated, while 
the actuators 72.2, 72.4, 72.6 and 72.8 are deactivated. 

In order to accommodate various layouts of control icons on the flexible display 50', as 
5 shown in Fig. 6, the actuators may be arranged in an actuator array 80 forming a matrix 
having a plurality of rows of actuators and columns of actuators. The rows of actuators in 
the actuator array 80 are addressed by a row interface 82, while the columns of actuators in 
the actuator array 80 are addressed by a column interface 84. An actuator controller 86 is 
then connected to the interfaces 82 and 84 for activating selected ones of the actuators 
1 0 corresponding to the location of the control icons as controlled by a display controller 88. 
With such an arrangement, small control icons may be raised using a single actuator, while 
larger control icons may be raised using multiple adjacent actuators. In fact, if the actuators 
are sufficiently small, a plural number of actuators may be used to form a raised 
distinguishable shape for the icon (e.g., an arrow), or may be used to form a type of rocker 
15 switch. 

While the invention so far has been described in the sense of forming raised areas on the 
surface of the display, it should be understood that, instead, depressions in the surface of 
the display by alternatively be formed. To this end, the pusher rods 74.1-74.9 of the 
actuators 72.1-72.9 are attached to the under-surface of the display 50'. Depending on the 
2 0 control signal applied to each actuator 72.1-72.9, the respective pusher rod 74.1-74.9 may 
press upwardly on die display (e.g., 74.1), remain in a neutral position (e.g., 74.2), or may 
pull down on die display. 

While the invention contemplates touch sensitivity on the part of the display, it should be 
noted that this feature has not been disclosed for the electroluminescent device of U.S. 
2 5 Patent 6,368,730. Hence, in order to provide for such, the actuator controller 86 may also 
detect pressure on, for example, the activated actuators, this pressure resulting from a user 
pressing the desired icon (note the two-way arrows connecting the interfaces 82 and 84 to 
the actuator array 80, and the two-way arrows connecting the interfeces 82 and 84 to the 
actuator controller 86). 

30 It should be noted that while the above description relates to the GUI layout as shown in 
Fig. 3, and in particular, to the bottom row of the GUI therein, the subject invention is 
applicable to other layouts. For example, the subject invention may be used to impart touch 
detectability to die display of the remote control unit shown in Figs. 1 and 2. In particular, 
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the small icons, e.g., 32, 34 and 36, may be accommodated by activating single actuators in 
the actuator array, while larger icons, e.g., 38, may be accommodated by activating two or 
more actuators. 

Disclosed next are two additional embodiments for providing tactile, or touch, detectability 
5 to Ihe GUI in the present invention. First, there is a vibrotactile display (i.e., at least one 
surface vibration) on the GUI; and, second, there is a electrotactile stimulation (i.e., 
electrocutaneous) on the GUI. 

FIG. 7 shows a side sectional view of a control panel 100 depicting the vibrotactile display 
embodiment (i.e., first type of tactile display). A plurality, or array, of cells 101 (i.e., 
10 101A-101G), which are connected via a plurality of pusher rods 102 (i.e., 102A-102G) to a 
plurality of actuators 103 (i.e., 103A-103G). The actuators 103 provide the means of 
vibration to the various cells 101 . In FIG. 7 three of the actuators 103D, 103E, 103F are 
activated and, thus, providing vibrations to cells 101D, 101E, 101F, respectively. The 
vibration of the cells 101D, 101E, 101F are depicted by directional arrows "V". Actuators 
15 101 can be driven by various means that can ultimately create at least one vibration in a 
respective cell such as piezoelectric crystals, small motors, etc. (not shown). 
Conversely, FIGS. 8A and 8B show side elevation sectional views depicting a portion of a 
control panel 100, and a control panel 100, respectively, with the second embodiment of 
providing tactile detectability (i.e., electrotactile stimulation). A plurality of cells 101 
2 0 make up a control panel 100 in the vibrotactile embodiment (See FIG. 7), while a plurality 
of electrode-skin interface cells 120 (i.e., 120A-120G) make up a control panel 100 in the 
electrotactile embodiment (See FIG. 8B). A single electrode-skin interface cell 120 (i.e., 
120A-120G) depicted in FIG. 8A typically includes an active electrode 122 (i.e., 122A- 
122G) situated between two return electrodes 121. The active electrode 122 is separated 
25 from the return electrodes by insulators 123 and air gaps 124 located above the insulators 
123. An electrical current 125 is provided to the active electrode 122. When a user's 
finger(s) 200 (shown in phantom in FIG. 8B) touches the active electrode 122 and 
concurrently touches one, or both, of the return electrodes 121 (i.e., closes the electrical 
circuit), the electrical current flows from the center active electrode 125 through the skin 
and subcutaneous tissue of tire user back through to one or both of the return electrodes 
121 . As a result, the user feels an electrotactile stimulation (i.e., mild electrical shock). In 
FIG. 8B three of the active electrodes 122D, 122E, 122F all are being provided with 
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electrical current thereby activating electrotactilely three electrode-skin interface cells 
120D, 120E, 120F, respectively from the control panel 100. 

FIGS. 9A and 9B bom show plan views of a portion of a control panel 99. In what is 
termed as a centralized-type (FIG. 9A) of vibrotactile display a portion of the control 
5 display (i.e., screen) 99 is made of a flexible material (e.g., plastic, etc.). In this manner, 
when an actuator 103 is activated to vibrate a portion of the control display 99, not only 
will the portion of the control display 99 immediately above the actuator vibrate, but the 
vibration effect will be transmitted in a whole adjacent area around an initial vibration 
point, as connoted by the vibration "node" 105. Alternatively, FIG. 9B depicts what is 
10 termed as a distributed-type of vibrotactile display. In the distributed-type of vibrotactile 
display, there is an array, or matrix, of cells 101. In FIG. 9B, the shaded cells 101L are 
each vibrating while the unshaded cells 101K are each not vibrating. An alternative 
embodiment of a distributed-type vibrotactile display is shown in FIG. 9C. Similarly, the 
shaded cells 101K are each vibrating, while the non-shaded cells 101L are each not 
15 vibrating. By varying the layout, pattern, or density, of which cells are each vibrating, or 
not, a different vibrotactile sensation, or feel, can be created. For example, although the 
perimeter of the vibrating area of bolh FIG. 9B and FIG. 9C is similar, the texture, or feel, 
of the vibrating areas in FIG. 9B and FIG. 89 will differ. 

Now wilh any of the preceding vibrotactile systems of FIGS. 9A-9C the vibrations can be 
2 0 pre-set or be programmable by an end-user. The vibrations can be set in an infinite 

combination of settings. By varying the characteristics of a vibration, the user can identify 
the function, control icon, or status of the control panel 100. Various characteristics mat 
can be changed include the texture, me frequency, the granularity, the duration, the 
amplitude, and the location of the vibration. 
25 The frequency of the vibration is the rate at which an actuator 103 activates to vibrate a 
particular cell 101, as in the distributed-type of vibrotactile system, or vibrate the control 
panel 100 as a whole, as in the centralized-type of vibrotactile system. Frequencies may 
run from about 30Hz to as high as about 1 kHz. 

The texture of the vibration includes the feel to the hand of the user of the vibration. For 
3 o example, in the distributed-type, the texture may be adjusted by varying how many of the 
cells 101 are vibrating and in what pattern the cells 101 are vibrating. This difference in 
texture of vibration is shown, and discussed above, in FIGS. 9B and 9C. 
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The granularity of the vibration can be altered. The granularity of the vibration refers to 
the smallest unit of vibration. Thus, each cell 101 may be the smallest unit of vibration. 
For example, the granularity can be between about 1 mm and about 5 mm. The granularity 
can be configurable on the device. 
5 The amplitude of the vibration would be the strength at which the various vibration means 
actuate (i.e., vibrate) a given cell 101 or area of the control panel 100. The amplitude can 
be programmable. 

The duration and location of the vibration can likewise be programmable. The duration is 
the length of time a particular vibration is maintained. The location of vibration is the 
10 exact location of cell 101, or cells 101, on the control panel 100 that is/are vibrating. 
Thus, by programming the various aforementioned vibrotactile characteristics to the 
control panel 100, a nearly infinite amount of patterns can be created. Similarly, various 
characteristics for the electro-tactile display can be programmed to create different 
sensations. 

15 For example, similar control icons on different display interfaces can have similar tactile 
characteristics. Also, the various control icons can have tactile characteristics, while the 
surrounding display area (i.e., background) has no tactile characteristics. Conversely, the 
display interface can be programmed with an opposite arrangement. That is, the control 
icons could have no tactile characteristics, while the surrounding areas of display exhibit 
20 the tactile characteristics. Additional features can be provided such as a screen saver 
whereby the tactile effect, be it electroctactile or vibrotactile, is off or at a lower energy 
while the control device is not being used. When the user picks up the control device, or 
touches a control icon or the control panel, the system is activated and the full tactile 
effects are maximized. This feature saves energy. 
2 5 FIGS. 10A, 10B, and 10C all show various embodiments of display interfaces (i.e., 

screens) of a control display 100. FIG. 10A, for example, shows a display interface of a 
control panel 100 utilizing the centralized-type of electrotactile display, in this case, 
depicting the various buttons for a TV display interface. FIG. 10B shows a display 
interface of a control panel 100 utilizing the centralized-type of electrotactile display, in 
30 this case, depicting the various buttons for a VCR display interface. FIG. 10C shows a 
display interface of a control panel 100 conversely utilizing the distributed-type of 
electrotactile display, again, depicting the various buttons for a VCR display interface. 
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Thus, by pressing a specific button on the display interface, the user can have the control 
panel 100 switch, or transition, between display interfaces (Le., screen displays). 
FIGS 1 1 A through 1 ID show a plan view of a control panel 100 with the progression of 
transitioning from the display interface for a TV (FIG. 1 1 A) to the display interface for a 
5 VCR(FIG. 11D). Various control icons 115 are depicted in the figures which serve to 
provide control functions to at least one device. When the user wishes to transition the 
control panel 100 from one display interface for a particular device (e.g., TV) to a second 
display interface for a second device (e.g., VCR), he/she presses a specific control icon 107 
which changes the display. The invention provides a particular tactile display, or effect, to 
1 o the control panel 100 so that the user knows, via feel, that the control panel 100 is 

transitioning from a first display interface to a second display interface. This transitioning 
effect can be accomplished via either the vibrotactile display or the electractile display. 
For example, this particular tactile display denoting the transitioning of the display 
interfaces could be a "wave effect". That is, as depicted in FIGS. 1 IB and 1 1C, like a 
15 wave, at least one vibration area 105 could spread across the control panel 100; hence, the 
term "wave effect". This "wave effect" could be a unique vibration moving across the 
control panel 100, from side to side; or, from top to bottom; or, some similar effect. The 
"wave effect" vibration area 105 could have different vibration characteristics than other 
vibrations on the control panel 1 00, thereby allowing the user to sense and know that the 
2 0 transitioning is taking place between display interfaces. A leading edge of the "wave 
effect" is denoted by directional arrows "W". 

Numerous alterations and modifications of the structure herein disclosed will present 
themselves to those skilled in the art. However, it is to be understood that the above 
described embodiment is for purposes of illustration only and not to be construed as a 
2 5 limitation of the invention. All such modifications which do not depart from the spirit of 
the invention are intended to be included within the scope of the appended claims. 
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